Live teleporting system and apparatus

ABSTRACT

A telepresence system includes a projector for generating an image, a projection screen for reviewing the image generated by the projector and generating a reflected image and a foil for reviewing the reflected image generated by the projection screen. The foil generates and directs partially reflected image toward an audience, the partially reflected image being perceived by the audience as a virtual image or hologram located on a viewing stage. Additionally, the system incorporates a camera for filming an individual through the foil, the camera being located on a camera side of the foil and positioned adjacent to the viewing stage, the individual being located on an individual side of the foil and positioned on a filming stage.

CROSS REFERENCE TO RELATED APPLICATION

This utility application is a National Stage Entry that claims priorityto “LIVE TELEPORTING SYSTEM AND APPARATUS,” having serial numberPCT/GB2009/050850, filed on Jul. 14, 2009, which claims priority to andthe benefit of U.S. Provisional Application No. 61/080,411, filing dateJul. 14, 2008.

DESCRIPTION OF THE RELATED ART

The origins and performance of video telephony are well documented. Insummary, the two-way interaction between two or more persons locatedremotely of one another is to a large degree dependent upon acommunications link and equipment capable of facilitating the desiredlevel of interaction.

The communications link may be a simple copper wire, a more substantialbroadband fibre optic cable, satellite or even radio waves.

In its most basic form, video telephony equipment comprises a handset ateach end of the connection, the handset being equipped with voiceprocessing and amplification for dialogue, a camera and video screen toenable the participants to see one another. Video telephones areincluded, as are personal computers equipped with the necessary webcams, telephony software and Internet connections.

More recently, the technology of Telepresence has been developed andrefined by companies such as Cisco and Teleris. Telepresence (TP) isdefined as a system of real time communication enabling two or morepeople located remotely of one another to exchange a dialogue based onthe principals of telephony (“Tele”) enhanced by the immersiveexperience of lowest time latency for a) high quality life size head andshoulders motion imagery of participants by way of using large(typically HD standard) video display monitors and b) maintaining eye toeye contact between participants during conversation c) complimented byintelligently lip synched audio, (“Presence”).

TP systems typically use the Internet communications cableinfrastructure as the signal conduit. Satellite can be used but whereasInternet connections allow two way or multiple way conversations withvideo to experience signal latency of as little as 40 milliseconds(acceptable as an immersive experience) satellite signals render latencylevels of 200 milliseconds or more, hence the experience forparticipants is one of perceptible signal delay. Radio waves offer lowlatency but the signal may be transmitted over only limiteddistances—typically up to just a few miles.

TP systems apparatus typically consist of video monitors to displayimages, audio equipment to record, amplify and broadcast voice/sound,cameras to capture the video images for display and codec enabling thesound and vision to be ‘packaged up’ (encrypted and compressed) in aformat optimised for point to point transmission between at least tworemote locations.

TP systems generally offer immersive experiences for between 2-18 peopleat one time. Generally one monitor is used to display head and shouldersof one person. However latterly, larger 65″ monitors or bigger are usedto display up to 3 different people sited in one location on a singlescreen. Monitors are grouped typically in a row array of 2, 3 or 4 unitsset along the wall of a conference room, facing a table as if thescreens are ‘sat’ at the table.

The object of this invention is to provide a more realistic or immersiveTP experience for use in larger and/or more public environments. Thisincludes providing TP in bigger meeting rooms allowing moreparticipants, TP for live stage environs (theatre, conference halls,museums, trade shows) and even in store and window front retaildisplays.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram depicting a codec and a broadcast streamof the prior art.

FIG. 2 is a schematic diagram depicting the internal workings of a codecunit of the prior art.

FIG. 3 is an illustration comparing various scan techniques of the priorart.

FIGS. 4-38 are schematic diagrams depicting example embodiments.

DETAILED DESCRIPTION

This invention comprises a number of enhancements to the entire set ofapparatus used in the TP process. Enhancements maybe used selectively oras a whole and thus the performance enhancements resulting maybe subtleor significant on a case by case basis.

Firstly, the use of a video panel screen for the display is the greatestlimiting factor for achieving both scale and an immersive experience.The reason being that monitors offer limited realism in the visualeffect whilst also consuming a greater amount of ‘data bandwidth’ toachieve their limited effect. Their limited realism comes about by wayof their video displays appearing as flat 2D images. This is common andwell known to any audience member familiar with watching conventionaltelevision screens.

TP vendors do attempt to mitigate the 2D effect by arranging the screensin rooms decorated in a uniform background to maximise the luminosity ofthe video subjects against their plain backdrop surroundings.Furthermore most recent TP systems use 50″ High Definition (HD) videomonitors, resulting in life size head and shoulders imagery that iscrisp with superior contrast.

Each monitor requires a camera located at the remote site to feed avideo signal for display and a microphone to record the sound or voicesignal (together the Signal Feed or SF). Each SF in turn needs a certainamount of data space, or bandwidth, from the communications link inorder to transmit the SF to the broadcast video monitor and soundsystem. The amount of data space required is in itself dependent upontwo key factors—the data size of the signal in its ‘unpackaged’(uncompressed) format and the way that SF is then ‘packaged’ orcompressed. The packaging of data is achieved using a video and soundcodec.

Codec accessories come in many forms. Generally a codec includessoftware for encryption and compression (together Encoding) of video andsound into a data packet which can then be transmitted over theInternet, satellite or radio waves. The codec is often incorporated intoa box chassis, much like the casing of a typical small network computerchassis. Codec chassis can have a variable number of inputs and outputsallowing the processing of multiple data streams or SF, inwards(downloading) and outwards (uploading). See attached diagram FIG. 1 tounderstand how a codec sits in the broadcast stream and FIG. 2 to viewinternal working of a codec unit.

Codec are designed and configured to process particular kinds of audioand video streams. This invention relates in the main to the most commonvideo streams of Broadcast Pal or NTSC (BP NTSC), High Definitionsignals of 720 horizontal lines progressive (720P), 1920 verticallines.times.1080 horizontal lines progressive (1080P) and 1920 verticallines.times.1080 horizontal lines interlaced (1080i). Other videostandards such as 2K and 4K resolutions could also benefit from theteachings here but we shall concern our solutions to be capable ofsolving the issues using video standards that are in widespread usecurrently.

In video, a field is one of the many still images which are displayedsequentially to create the impression of motion on the screen. Twofields comprise one video frame. When the fields are displayed on avideo monitor they are “interlaced” so that the content of one fieldwill be used on all of the odd-numbered lines on the screen and theother field will be displayed on the even lines. Converting fields to astill frame image requires a process called deinterlacing, in which themissing lines are duplicated or interpolated to recreate the informationthat would have been contained in the discarded field. Since each fieldcontains only half of the information of a full frame, however,deinterlaced images do not have the resolution of a full frame.

In order to increase the resolution of video images, therefore, newschemes have been created that capture full-frame images for each frame.Video composed of such frames is called progressive scan video.

Progressive or noninterlaced scanning is a method for displaying,storing or transmitting moving images in which all the lines of eachframe are drawn in sequence. This is in contrast to the interlacing usedin traditional television systems where only the odd lines, then theeven lines of each frame (each image now called a field) are drawnalternatively.

The system was originally known as “sequential scanning” when it wasused in the Baird 240 line television transmissions from AlexandraPalace, England in 1936. It was also used in Baird's experimentaltransmissions using 30 lines in the 1920s. (The system may also becalled 240p25 and 30p25).

This illustration of FIG. 3 compares progressive scan (480p) withinterlace scan (480i), also demonstrating the interline twitter effectassociated with interlace. The interlaced images use half the bandwidthof the progressive ones. The left-center image precisely duplicates thepixels of the progressive one, but interlace causes details to twitter.Real interlaced video blurs such details to prevent twitter, but as seenon the right-center image, such softening (or anti-aliasing) comes atthe cost of image clarity. A line doubler could not restore thepreviously interlaced image on the right to the full quality of theprogressive image on the left.

1080p

1080p is sometimes referred to in marketing materials as “CompleteHigh-Definition”. However, 2K/4K digital cinema technology iscommercially available, and ultra-high definition video is in theresearch phase.

Broadcasting Standards

ATSC and DVB support 1080p video, but only at the frame rates of 24, 25,and 30 frames per second (1080p24, 1080p25, 1080p30) and their1000/1001-rate slow versions (e.g., 29.97 frames per second instead of30). Higher frame rates, such as 1080p50 and 1080p60, could only be sentwith more bandwidth or if a more advanced codec (such as H.264/MPEG-4AVC) were used. Higher frame rates such as 1080p50 and 1080p60 areforeseen as the future broadcasting standard for production.

A new high-definition progressive scan format is not available forpicture creation, but is currently being developed to operate at 1080pat 50 or 60 frames per second. This format will require a whole newrange of studio equipment including cameras, storage, edit andcontribution links as it has doubled the data rate of current 50 or 60fields interlaced 1920.times.1080 from 1.485 Gbits/sec to nominally 3Gbits/sec.

Image Change Rate

There are several agreed standard image change rates (or frame rates) inuse today: 24 Hz, 25 Hz, 30 Hz, 50 Hz, and 60 Hz. Technical detailsrelated to the backwards-compatible addition of colour to the NTSCsignal caused other variants to appear: 24/1.001 Hz, 30/1.001 Hz,60/1.001 Hz.

The image change rate fundamentally affects how “fluid” the motion itcaptures will look on the screen. Moving image material, based on this,is sometimes roughly divided into 2 groups: the so called film-basedmaterial, where the image of the scene is captured by camera 24 times asecond (24 Hz), and the video-based material, where the image iscaptured 50 or ˜60 times a second.

The 50 and ˜60 Hz material captures motion very well, and it looks veryfluid on the screen. The 24 Hz material, in principle, captures motionsatisfactorily, however because it is usually displayed at least attwice the capture rate in cinema and on CRT TV (to avoid flicker), it isnot considered to be capable of transmitting “fluid” motion. It isnevertheless continued to be used for filming movies due to the uniqueartistic impression arising exactly from the slow image change rate.

Codec works by taking a video signal that, for example, from a 1080icamera filming at 50 frames per second via an HD SDI connection SMPTE292m (Society of Motion Picture Technicians and Engineers recognisedvideo standard) is typically 1 485 megabits (m/bits or MB) of data persecond and compressing that signal sufficiently to allow transmissionalong a broadband Internet line.

Internet lines in themselves are of varying capacity. Typical consumerlines are 2-8 megabit speeds. Business lines vary from consumer typespeeds, to E3 (34 m/bits), DS3 (45 m/bits) 155 m/bits and beyond.However the speeds are not constant, rather are a maximum.

By way of comparison conventional road standards offer a good analogy. Ahighway with a speed limit of 70 miles per hour may well indeed allowspeeds of 70 MPH to be maintained when traffic on that highway is light.However a number of factors, not least of which a heavy traffic load,conspire to render achievable speeds averaging far less.

The caveat in using this road analogy is that whereas one can set ahigher average speed for a road journey by arbitrarily exceeding speedlimits in light traffic to compensate for slower speeds whereinterruptions occur, on the data highway no such compensatory factorsusually apply. The speed of the line is the maximum speed of the lineand thus if data interruptions do occur (which are inevitable on apublic Internet connection) the signal is affected irrevocably.

Moreover, when one undertakes a road journey one does not necessarilyhave 70 MPH highway from destination to destination. Rather there aresmaller roads with slower speeds. So it is with the public Internethighway. A fast 10 MB public line may bottleneck at some point beforereaching its destination, thus again affecting the signal which, in TPapplications, manifests itself as a sound/video/picture drop out—i.e. atemporary blank screen or a blast of missing words—unacceptable for arealistic immersive interactive experience.

Current TP solutions are for codec to compress SF as small as practicaland for cable networks to be managed so they run as consistently fast ascommercially viable bandwidth allows. The result is that for codec,generally a raw BP NTSC signal compresses from around 550 m/bits to1.5-2 m/bits per second, 720P to between 2-3 m/bits per second and 1080signals to between 4 and 8 m/bits per second. TP networks are typically10 MB for carrying up to 4 SF of either BP NTSC or 720P systems, E3standard if more screens or 1080 signals are used and DS3 lines incircumstances where there are more than 4 screens requiring 1080streams.

Signal consistency through the cable is provided by way of a managed orVirtual Private Network (VPN). This means that the SF data flow (uploadand download) requires a dedicated line capable of guaranteeing thenecessary minimum m/bits transmission speed consistently. For this TPusers must pay a financial premium to have either a network built fortheir exclusive use (a Private Virtual network) wherein a technicalmanager be usually employed by that user.

Once the SF has been delivered to its broadcast destination anothercodec decompresses the SF at the point just before broadcast to thevideo monitor and sound system.

The effect of codec compressing signals to a fraction of their raw datarate during transmission is that certain elements of the original data'sintegrity is lost or in some way degraded. Compression of an HD signaltypically causes dilution of image colour saturation, reduced contrastas well inducing motion blur around the entire body down to apparentloss of lens focus on details such as eye sockets, or where the videoimage has high contrast.

Conventional TP systems are not adversely affected by the compromises tovideo quality since it is a design fundamental to seat participantsaround a conference table using 50″ monitors for broadcast—a settingsuited to minimal subject movement and colour/contrast variationsrestricted to head and shoulders only against a uniformly decoratedbackground.

A TP system optimised for broadcasting images over areas of 24M2 of morerequires a different approach to all the elements of apparatusperformance or use.

Firstly, the use of monitors maybe suitable for conference roomapplications. However video signals to 50″ plasma monitors are notsuitable for displaying life size human images. One solution is to uselatest generation HD 103″ monitors. If arranged vertically thesemonitors could display full size figures from a single SF. However, theimage still suffers from the limitations inherent to using monitorswould look flat and be incapable of any lateral movement on account ofthe limited screen width of just 24″. Large monitors are also currentlyvery costly. Moreover if a conference required multiple figures toaddress a watching audience then a monitor would be needed for eachfigure, requiring further expense of multiple monitors as well as morecodec, greater Internet bandwidth and potentially higher IT costsoverall.

A more practical solution would be to use a semi transparent foil screensecured under tension within a frame to form a smooth, flat surface,configured with a reflective front or rear projection screen andamplified light source to display video images in a ‘peppers ghost’arrangement. This arrangement provides a number of well documentedadvantages in the field of video presentations upon a stage. Whencombined with TP, the use of a foil and projection adds a furthersignificant advantage.

Peppers ghost images of ‘virtual’ human beings are becoming ever morerealistic with advances in foil screen manufacture and installationprocesses allowing reflective polymer foil material as thin as 11microns to form large screens with surface areas typically up to 36 mwide×8.1 m high characterised by surfaces that are smooth and free fromsurface deformities such as creases or wrinkles. The result is a screenthat when used as part of an illuminated stage apparatus is all butinvisible to the viewing audience yet is capable of ‘bouncing’(reflecting) imagery (solid or video) onto the stage that is virtuallyindistinguishable from the image of the original.

The advances in foil preparation are further complimented bydevelopments in film capture (cameras, lighting and set design) andbetter broadcast technology (projectors' resolution and brightness).Companies like Musion Systems have refined the art of ‘virtual’ imageryso that video projected onto a stage using a foil is virtuallyindistinguishable from the original in terms of visual likeness.Musion's technique relies upon a number of production and apparatusprocesses including optimising the projection throw of the projectionlens according to the distance the viewing audience are from theprojected image.

Using a foil, a typical HD projector of 10 000 lumens brightness and1920×1080 pixels can project realistic images of virtual human beings orother objects up to 5 m wide, provided the optimal viewing distance ofthe audience viewing is at least 5 m distance away. Should the viewingaudience be less, the throw of the projector would be shorter (or anarrower throw lens used), rendering the pixel count tighter and theimage would be correspondingly shrunk—ideally to the optimal 3 m widthfor 3 m viewing distance.

Pixel size for the width of 5 m for a 1920×1080 image of projectedpixels is significantly greater than the 1 m width or so a plasma screenoffers for its 1920 pixels. The result is that for a single SF, a userdeploying a 16×9 HD projector can utilise a viewing area of 5 mwidth×2.8 m height—sufficient to be viewable upon a stage area typicalof conferences, trade shows and other live audience displays.

Furthermore if the projector is arranged within a 6 m×4 m foil screenbased peppers ghost apparatus as disclosed in Maass U.S. Pat. No.5,865,519, not only are the realistic looking SF video images broadcastwith the means of video the sources for which are invisible to theaudience, but live presenters may interact with these virtual figuresand imagery on the same stage, or in the same field area. Theseembodiments add considerably to the realism or immersive experience ofTP.

However, monitors aside, TP systems designed for board rooms docompromise the realistic effect by causing displaying motion blur whenvirtual figures undertake any left/right movement. Current TP codecsystems shrink an HD SF from 1.5 gigabits/per second to just 5 or 6megabits/per second. Non TP codec systems compress a signal less. Codecused in sports broadcast (where higher speed motion is more commonplace)compresses a signal to between 20-30 megabits/per second. However thetrade off is that such large signals used in TP would result in greaterlevels of signal latency of the SF thus rendering the realism ofimmediate low latency interaction between SF and audience or live stagetalent. Moreover, the need for 30 mega/bits for a single SF wouldrequire significantly higher available network bandwidth.

One solution would be to limit the movement of SF. This would not bepractical if the SF is an entertainment artist or presenter whoseperformance relies on expressive movement. A compromise would be to usea variable bit rate codec capable of being switched from low bit rate incircumstances where low latency response times for SF/live interactionis crucial (such as Q&A sessions) and higher bit rate when motion videoquality is important and actual SF interaction is not required (duringan actual performance or presentation). In the latter element, bandwidthotherwise assigned to the interactive elements of the show (audiencecamera/s, positional reference camera/s) could be switched temporarilywhilst not in use, concentrating all the available bandwidth instead ondelivering the most realistic image experience.

This is most conveniently achieved using a switchable scalar (Spyder orEncore) along with the associated equipment. At the press of acontroller button (managed either by presenter, artist or otherdesignated show controllers) at appropriate moments during apresentation or performance. The control button is linked to a networkrouter managing the codec download/upload data feeds (SF).

Another challenge for performing a realistic TP experience upon a stagerather than around a table is that of real time positional referencedevices. FIG. 4 shows the positioning of monitors/screens for:

a) Broadcast Stage (BS), live talent (Compare) interaction with TPvirtual figure (Performer) from SF Source Stage (SS).

b) SF Source Stage, the virtual Compare interaction with live PerformerSF-SS.

c) Live Performer on SF Source Stage interaction with TP audience facingbroadcast stage.

d) Heads up display teleprompter on BS to reference live Compare orPresenter for realistic positional stance and eyelevel relative tospecific audience areas, whilst enabling Compare or Presenter to readtext or any other video image during Performance. This includes a videoimage of audience members selected by audience cameras (see camerasbelow).

For the purposes of invention, Compare could also include multiplecharacters interacting with the TP virtual figure/s. The solutionsproposed can be scaled as multiple feeds of virtual figures via TP fromBS to SS or SS stage to BS.

The conventional methods of positional reference for existing TP is touse large HD 1080 plasma or LCD monitors with HD cameras attached. Themonitors thus serve the dual purpose of capturing and displaying the TPvirtual images from either stage/s. The monitors are positioned stageleft or right out of view of the audience, viewable to the Compare andPerformer (C&P). Their object is to provide real time eye to eye contactbetween (C&P), reading of C&P body language and positional referencerelevant to each others' stage position.

Limitations are that the viewable movement of the C&P is restricted toeither/and the capabilities of the camera lens frame of the monitorscreen size. Movement up-stage or downstage is particularly affected ifthe monitor is arranged vertically since a 16×9 screen on 65″ monitormeasures just 60 cm along its narrower edge. Moreover the camerasattached to the monitors filming the TP SF are conventionally designedto provide images filmed in the conference room environment—controlledlighting and limited participant movement seated around a table. Thustheir lenses are not optimised to best capture the C&P proceedings in alarger stage type area, lit in a more theatrical fashion.

Monitors could be replaced by conventional front or rear projectionscreens displaying the TP signal by projection. This solution ispreferable to the use of multiple monitor panels for technical and costreasons earlier explained. From a practical point of view a bank ofvideo screens on stage is also cumbersome to assemble or locateunobtrusively.

The conventional projection carries certain disadvantages however—notleast the positioning of the projection path. For optimal referencingthe base of the monitor or, in this case, the projection screen shouldcorrespond with the base of the stage. A projection path arranged tolight a large screen around the base of a stage entrances is a hindranceto live talent or indeed back stage support staff operating around thesecrucial show areas, regardless of whether the screen is front lit orback-lit.

Another alternative is the use of a smooth, semi transparent, antistatic, easy clean fire retardant foil tensioned within a frame arrangedin a number of different fashions using a variety of video sources.

The Foil frame could be made of a lightweight polymer, steel, carbonfibre or aluminium suitable for easy stage flying, so it may be attachedto a conventional stage flying system or simple movement motors forstorage flat in a ceiling. The frame carries adjustment allowing thefoil to be re-tensioned repeatedly for a wrinkle free and flat surfacefinish during operation. The foil's anti static easy clean finishassists in retaining the foil's smoothness and optimal screen clarity.

The Foil frame could be attached to a front of rear projection screen ina variety of configurations known as peppers ghost shown in FIGS. 5-22.The FP/RP screens display source video images from projectors(preferably 1080 HD), or the foil directly from video or LED walls,which too can be attached to the Foil frame. This is shown in FIG. 23.

The frame could be attached directly to a plasma or LCD monitor. Using a103″ plasma hung so the screen facing towards the foil reflects an imagethe other side of the foil—is big enough to display full size humanbeings as a single codec signal using a drastically reduced projectionpath distance compared to projectors and bounce screens.

The monitor may be located facing upwards towards a foil angleddownwards at 45 degrees, fully horizontal under a clear protected stagefloor. This configuration allows the stage floor to be used in an areathat would otherwise be reserved only for beam through onto the foil.

Another embodiment is an adjustably moveable stage set—partiallysubmerged, more vertically inclined in the stage floor, the screen,frame and monitor disguised from audience or TP source talent vieweither as invisible black or as a component of stage set/scenery.Similarly, the Foil frame and monitor could be flown and operative aboveand away from the potentially busy stage entrance.

In summary, the use of an HD screen monitor of sufficient picturequality, brightness and size, positioned to screen its image as apeppers ghost through a foil creates significant extra stage floor spaceand convenience to the live talent performing on and around the stageand stage entrances.

Alternatively, the Foil frame could be isolated to work with videosources projecting remotely to it. This arrangement is shown in FIG. 23.In this embodiment powerful but heavy projectors or LED wall areinstalled in more permanent fixings above or below the stage set, theirchassis and projection path positioned to avoid interference with themovements by live on stage talent.

All of the above configurations also provide a further advantage ofpositioning the camera lens for TP SF capture. By using a monitor panelor conventional projection screen the camera lens must be located aboutthe periphery edge of the display. Using a transparent foil allows thecamera to be positioned anywhere, including directly behind the screenas shown in FIG. 24. The foil if correctly prepared during installationshall have a smooth uniform surface that does not impede the lens viewof the TP camera allowing images to be captured by shooting through thefoil. Moreover the appearance camera side of a virtual image visible tothe live talent or audience also does not affect the lens viewwhatsoever.

This feature is of practical use for providing line of sight guidance totalent requiring accurate eye to eye contact during two way real timevideo communications and/or in the design of a TP meeting room or areaof limited size. The live talent upon either the SS or BS is able toview ‘hard copy’ references through the transparent foil screen as wellas the virtual image. Such a hard copy reference could be a light orsignal designed to accurately guide the precise direction of eye view.

Another Foil arrangement is of particular advantage for displayingimages as part of an enhanced TP system. The image, instead of appearingupstage (on the opposite side) of the foil to the C&P or live viewingaudience, rather, appears some distance in front of the foil with noscreen separating the virtual image from the live talent. This allowsmore accurate latitudinal positional reference (fore and aft depth offield upon a stage between the Compare and Performer, left and rightpositional reference relative to an audience).

The art of using parabolic reflective mirrors for display of virtualimages is known. The image when viewed from a certain angle (albeitlimited to a rather narrow front-on viewing angle) nevertheless appearsto be floating in mid air. If the source image is 3D, then the floatingimage appears 3D too, even when static. The limiting factor forparabolic mirrors is the size of the mirror itself—being limited to thelimited size of casting tools used in the parabolic ‘bowl’ or mirrormanufacture.

The distance between position of the virtual image and the foilapparatus (i.e. how far image appears in front of the foil) isdetermined by the depth of the mirror or bowl's concavity as well as itsactual size as well as the distance between the source video screen andmirror centre. FIG. 25 shows parabolic bowls of variable shapes andsizes, configured with a direct video source such as a monitor or LEDscreen as well as a projector beaming onto a rear projection screendirected towards the central area of the parabolic.

To be effective in delivering a highly realistic and immersiveexperience, the virtual image should be HD video and if a human figure,life size. Regrettably current solid polymer moulds capable of use asparabolic mirrors and commercially available tend to be a maximumdiameter of 2-2.5 meters and a depth 1-1.5 m. The image size achievableusing practical projection means is a maximum of between 80 cm-1 m—notsufficient for displaying a life size virtual human.

One solution is to use a Foil vacuum blown to form shape in purposebuilt housing, to adopt and then retain the shape used in the cast ofparabolic reflectors. This is achieved by tensioning a foil verticallyat the front of the structure, much as a screen faces a monitor. Thefoil edges are sealed so that a vacuum applied to the box behind thefoil eventually sucks the foil into a pre determined ‘parabolic’ shape.The foil depth of concavity could be varied according adjustment usingvariable vacuum pressure.

The foil shape could be retained in operation by vacuum but another,more desirable method of forming from foil a parabolic shape forpermanent use is to simultaneously feed a solidifying liquid substance(such as liquid polystyrene foam used for extinguishing fire) into thebox whilst the foil's correct parabolic shaped is retained under vacuumsuction. Once the foam solidifies, the shape remains permanent. The foamis coloured black so once set the foil surface appears to be a shinyreflective black parabolic mirror.

The advantage of using foil is not just the ease of construction andpotentially lighter weight than a solid polymer bowl of similar size.The key advantage is size. Optically clear/semi transparent fireretardant antistatic foil of widths 6-8 meters would provide thenecessary sheet size enabling a parabolic mirror to reflect virtualimages sized up to 2 m high.times.2 m wide.

Various configurations are suited to a variety of applications where TPinteractive figures would be of benefit. For stage applications themirrors may be used effectively face on to the live stage talent. Theangle of view is such live talent can simply reference themselves tointeractive virtual images for spatial distance as well as left rightmovement over reasonably large stage areas, yet the audience does notsee the virtual image emitting from the foil parabolic mirror.

In other applications such as retail shop windows or museums theparabolic mirror is arranged face on the viewing audience.

Conventional TP cameras attached to monitors do not offer featuresrequired for the most immersive of TP experiences. The signal image hasa very limited positional and movement sweet spot to capture realisticvirtual images—in a narrow width band of 1.5 m and a maximum distancebetween lens and subject of up to 8 feet (2.5 m). The cameras do notcarry adjustable iris that could be electronically controlled to adjustfor variable lighting conditions (as might be found on a theatricalstage or used in filming subjects of different textures and colours).The absence of iris adjustment results in images looking too dark orappearing bleached out. Incorrect or uneven lighting renders the TPimages somewhat unrealistic. For certain image colour types (pure redsor colours like brown that include reds) insufficient lighting levelsalso cause motion blur. Correctable using brighter lighting arrangementsand faster shutter speeds.

The lack of zoom adjustment (mechanical, electronic or otherwise)restricts the field area the lens can film thus restricting cameraplacement (to the busy sweet spot of the stage entrance). The lensesfound on existing TP feature a zoom insufficient to film an area anybigger than a 103″ monitor. Restrictive movement reduces realism,particularly for performances of moving entertainment whilst introducingfor C&P unwelcome movement boundaries that may hinder performancespontaneity generally.

By providing for the camera to shoot effectively through a transparentfoil from some distance behind the screen, camera placement is moreflexible. A camera able to work with a variety of professional zoomlenses, with adjustable shutter speeds and multi-video standards wouldbe more effective in capturing a greater field area of view lightingthus allowing Performers greater freedom of movement and spontaneity.

A common standard HD Cam are the Sony models HDW X750, HDW 790, F900R,all of which are single link HD SDI processing 10 bit 422 colour streamsat 1.485 Gigabits/per second and F23 which is both a single and duallink HD SDI processing 12 bit 444 colour streams at 2.2 Gigabits/persecond. These models yield finest picture results using the HD SDIsignal at 50/60 frames interlaced per second. Progressive camerasinclude the Panasonic AJ-HDC27HE 720P Varicam and Red Camera—capable of4K resolution. However because of Progressive signal's higher datacapacity demands per frame versus interlaced, it is common for thesecameras to film at just 25 frames or in USA, 29.97 frames Progressiveper second. TP at 50 frames Progressive HD is not yet commerciallyavailable as the signal data rate is too great for conventional codec tomanage and compress.

Progressive signals offer sharpness, particularly for static images whencompared with interlaced video, which tends to have softer, less crispedges. Conventional TP systems display seated images upon a 50″ screen.Movement is limited, thus TP codec using 1080P as the video standard(such as Cisco TP) images appear sharp. Progressive HD is less wellsuited to handling bi-directional subject movement. This is because atthe same frame rate, progressive video will use twice the bandwidth ofan interlaced signal. The result is that an interlaced signal (typically50i/60i-50/60 frames per second (fps)), though only half the horizontalresolution of a 50/60 fps progressive, has more moving fields per secondwhich to the eye appears to offer a smoother motion at the same framerate. This is because for the process of displaying a field of video aninterlaced signal is consuming only half of the progressive signal'sbandwidth.

Moreover Progressive HD looks less realistic than an interlaced signalwhen used to display realistic looking virtual humans using the foilpeppers ghost technique. Progressive HD images appear flatter to theviewing audience—like 2D film images. This is because the image iscomposited as a whole, rather than being interlaced. Interlaced HDsignals of at least 50 frames per second are ideal as a video standardfor motion virtual images using foil projection. This is because aninterlaced signal of 50 frames per second has twice the time basedfrequency compared to progressive 25P (despite the same bandwidth). Textand graphics, particularly static graphics, however benefit from beinggenerated using a Progressive signal of at least 25 frames per secondbecause progressive displays the complete frame of video in one unit oftime (every 25^(th) of a second) versus interlaced showing just half acomplete frame every 50^(th) of a second resulting in a progressivesignal effectively doubling the horizontal resolution to form smoother,sharper outline edges for static images. Thus a camera used for TP humanfilming should be HD-SDI enabled. The frame rate speed should be higherthe faster the movement. Hence certain HD cameras used for filming sportcan run up to 500 frames a second. For dancing performance and othersudden movement scenarios, an HDSDI signal at 120 frames per secondwould be most ideal. The data rate requiring real time encoding(compression) would be higher than 50 or 60 per second, but the finalcompression to SF codec would be 20 M/bits per second. High speed framerates would therefore be transmitted via codec using the pictureoptimised encode. Slower film frame rates are engaged for Presenterswith less movement.

In summary a camera utilising a light sensitive high quality wide anglezoom lens with adjustable shutter speed, frame rates adjustable between25-120 frames per second (fps) interlaced, capable of shooting at up to60 fps progressive, would address the key range of performancerequirements for most kinds of video imagery, from static texts andgraphics to streaming images of virtual Presenters and even movementartists.

It would be desirable for the camera to have a remote moving headattached to a ‘magic arm’, allowing motorised mechanical movementanchored to a convenient mounting position. It would be desirable forthe camera's features and adjustments to be controlled remotely via LANand to programmable to environmental pre sets (such as shutter speedresponding to programmed subject matter/lighting inputs).

The camera's position varies according to its function within the TPSystem. If the camera is to provide the SF for an on stage virtualPerformer, the lens position relative to the live Performer shouldcorrespond to the eye line view of the watching audience as shown inFIG. 26. The Source Stage Performer feed to Broadcast Stage is perhapsthe most important task to overcome challenges in order to optimisevisual quality. As the main subject of the stage show the Performer willbe the key to determining show realism overall.

The appearance of depth up stage/down stage is an illusion. Thisillusion is most effectively performed when the audience eye line isjust below the line of the stage floor and the camera lens filming thelive Presenter is positioned at least 5 m away from the subject andangled corresponding to the angle of audience view. By way of example,the angle of view is ideal when the viewing audience is able to witnessglimpses of the shoe soles belonging to the virtual Performer as he orshe walks about the stage. However it should be noted that incircumstances where the camera view is shooting through a foil in thedesign shown for a pair of mirrored TP rooms, then the lens view mustclear the ancillary masking of the projection pit enclosing thereflective ‘bounce’ screen.

Raked audiences benefit from viewing Performers upon stages that arealso raked to the corresponding angle. See claims 11-14 Musion PatentApplication GB 0625525.1 Video Shadowing concerning arrangement of astage using a height adjustable audience chair.

The frame size of a camera lens determines the field area sent as SF.Broadly speaking, in order to optimise field area view and realismquality of subject matter within the frame size—referred to hereon asPlate Shot, should correspond to the audience viewing distance in a waysimilar to projection lens throw described earlier. Once the Plate Shothas been determined, the camera is ‘locked off’ i.e. the chassis doesnot move during operation.

Thus an audience seated 5 m or greater apart from the virtual image canbe filmed using a 16×9 Plate Shot size of 5 m width×2.8 m high—more than4 times the field area coverage of a conventional TP camera feeding theTP codec the same sized signal to a monitor screen.

For extra image solidity and sharpness both the Plate Shot andprojection throw can be limited to a smaller size for example 3 mwidth×1.7 m high—thus maximising the projector's brightness into asmaller concentrated space and the 1920×1080 pixel panel used in formingthe image of say 1.68 m high. This technique is particularlyadvantageous when a presentation or performance necessitates filming ofthe TP virtual figure on stage for real time video relay to large imageside screens. The denser pixel count and brighter image looks more solidand realistic when enlarged to bigger side screens. This technique mayalso be used where bandwidth restrictions dictate HD images areprojected using codec compression as low as 3-4 M/bits per second.

The next camera is positioned somewhere upon the stage pointing towardsthe audience. The objective of this camera is to provide virtualPerformer with maximum visual general audience feedback and optimise eyeto eye contact between Performer and audience members, collectively orindividually. The precise location is dependent upon a number offactors.

Overriding consideration should be given to optimising lens position tofeed the clearest possible audience view to the live Performer on SS(virtual Performer on BS). Furthermore the camera view should ifpractical replicate as accurately as possible the perspective view ofthe BS virtual Performer relative to the audience—thus positioning thelens at approximate eyelevel is desirable. In the event a large audienceis present more than one camera may be used. Indeed for large audiencesa number of solutions are available.

The first solution is to mount a remote head camera or multiple remotehead cameras using magic arms, enabling these cameras to move whilstanchored to a mounting point. The cameras could be equipped withlighting integral to the chassis, to assist in the lighting of filmsubjects. The cameras are equipped with variable zoom facility enablingremote adjustment in fore/aft range of at least 10 m. The cameras arecapable of remotely adjustable iris to light intensity and adjustableshutter speeds allowing their speeds to be varied as a means of reducingmotion blur in the filming process. The cameras are enabled to processeither progressive or interlaced HD video signals. For displaying seatedaudience images a progressive signal is desirable. The cameras may befitted with microphones enabling voice recording in real time. Thecamera may be enabled to recognise and track a signal or object (such asan infra red or ultra violet light, or a black and white patternedbarcode). Once the lens registers the signal, pre programmed settingsdirect the camera's view.

Thus in an audience of hundreds or even thousands of people, when anaudience member is chosen to interact in real time with on stagePerformers or Compares (live or virtual) an audience management systemmay be used that highlights in a way recognisable to the camera lens theprecise position of that audience member. The program control of thecamera would enable the zoom lens and any additional light or soundrecording devices to focus predominantly on the audience member, feedingback an image to the live or virtual Performer that is clear andreferentially accurate in terms of eye line.

The light or sound recording devices used for audience members may bepre set. Lighting is permanently installed and powered on to light theaudience/individual audience members whenever needed. Cameras andmicrophones arranged likewise. FIG. 27 shows how an auditorium might beconfigured for light, camera and sound. FIG. 28 shows how a smaller TPmeeting room might be configured. The arrangements show lighting andsound recorders arranged throughout the auditorium. Lighting is angledtowards the audience and away from the stage so as not to feed back asmuch audience vision as possible to Performer whilst not impedingaudience vision and experience of the foil projected images.

Another solution would be to use audience seating arrangements that werebetter suited to facilitate interactive experiences between bothremotely located TP virtual Performers appearing on stage by way of foilprojection and live Compares or Performers appearing upon a stage wherefoil projection is operative. Each audience seat block or individualseat would be equipped with devices enabling the audience member totable interest to interact with the stage talent—e.g. to ask aquestion—such that when selected, the seating area around the audienceparticipant is then automatically lit for optimal motion video imagecapture, a nearby sound recording device and remote head camera (locatedon a magic arm either individually to each seat or seat block) activatesto begin transmitting a suitable Audience Signal Feed (ASF).

The ASF can be then routed to one or more reference screens locatedeither upon the BS or SS. The BS screen may be a conventional monitorpanel located in the front of the stage (projection pit) to face thelive Performer or Compare directly. The SS screen may also be aconventional monitor panel located close to the lens of the SF camerafacing the SS live Performer (virtual Performer on BS). However adrawback to both scenarios is the screen being anchored to a fixedpoint, the positional reference for which may bear no resemblance to thepositional reference between the audience participant and the livePerformer or Compare on either stage.

The screen located at the BS may be mechanically moveable stage left orright according to angle of audience participant relative to the stagetalent. However, whether placed in front of or above the stage, theaudience participant image could not be satisfactorily positioned ateyelevel to the live stage Performer or Compare, without the monitorframe being visible to the audience. This would be a distraction to theimmersive experience sought.

This problem is most conveniently solved by mounting a mechanicallymoveable video screen/s upstage to the foil, the foil is inclined at anangle with respect to a plane of emission of light from an amplifiedlight source (projector using RP or FP, LCD, LED or powered lights); thefoil having a front surface arranged such that light emitted from thevideo screen is reflected therefrom; and the video screen being arrangedto project an image such that light forming the image impinges upon thefoil upstage of the audience (and thus invisible to Audience) such thata virtual image is created from light reflected from the screen, thevirtual image appearing to be located behind the screen, or down stageof the Presenter.

The video screen may be a conventional LCD/TFT monitor panel. It may beattached to the stage truss framing the foil screen and positioned in asubstantially horizontal fashion, screen angled downwards towards thefoil (in a similar way to the 103″ panel fixed to the foil describedearlier). The video signal from the monitor appears to the livePerformer or Compare as a peppers ghost image floating directly above oramongst the audience participants, yet the images are entirely invisibleto the audience. This is similar to the earlier principal of a camerashooting through the foil even whilst a virtual image is masking thecamera's presence.

The monitor/screen may be mechanically moveable stage left or rightalong the truss using rollers. The angle of the screen may be remotelyadjustable to position the reflected peppers ghost image higher or loweraround the audience. The position of the monitor/screen may bereferenced to the position and eye line angle between the live Performeror Compare on stage, relative to the position of the audienceparticipant. The image upon the screen may be a close up camera shot ofthe audience participant creating the illusion of a virtual audiencemember, highlighted or enlarged as a peppers ghost image appearing inthe same seating block/seat as the live audience member. The use of anASF through the foil brings the communicating parties ‘closer’ enablingthe live on stage Performers and Compares to experience a facial detailand intensity of audience interaction (including eye to eye contact) notpreviously possible.

More than one screen may be used so that the range of motion along thetruss does not impinge on the operation of other video or lightemissions concurrently operative with the foil. The immersive impact ofthis effect is greatly enhanced for audience participants if the onstage talent is filmed from a downstage location, the images beingtransmitted real time to larger relay screens located either side of thestage or to the side of or above the audience areas generally. Thisarrangement allows better body/facial detail of the live talent to beseen by the audience during performances.

The video screen facing the live performer in the SS could also be aconventional 50/60/82/103″ monitor panel, positioned close to the cameralens filming the SS live Performer for relay as the virtual performer onthe BS. This arrangement is practical where the audience numbers aresmall, each audience participant being clearly visible at the same time.

For larger audiences or circumstances where the camera lens filming theSS live performer is some distance away (say greater than 5 m) an imageprojector beaming the ASF/s onto a rear projection screen (RPS) locatedjust above the camera lens is desirable. The RPS can be of any size, butto offer greater utility than a monitor should have a surface area of atleast 3 m×2 m, arranged vertically or horizontally according to theshape of the audience viewing area and the frame of the camera lensfeeding the ASF/s. Preferably the projector will be a 1080 HD, capableof processing, both progressive and interlaced signals respectively,through DVI/HDMI and HDSDI interfaces built into the projector.

The ASF/s appearing on the RPS could consist of one or more images. Oneimage could be of the entire audience fed from the on stage TP camera.Another image could be of the individual audience participant fed froman additional camera positioned above or within a specific audienceseating block close to the participant. Using a suitable videoprocessing device such as an Encore or Spyder, the video image couldappear upon the RPS as a scalable picture within picture—that is animage of the participant within the image of the entire audience. Toassist with maintaining eye level contact, the precise position ofparticipant image upon the RPS could be referenced to the participant'sposition in the audience, relevant to the onstage position of virtualPerformer appearing on the BS.

Another solution would be to arrange the SS camera filming the livePerformer SF to the BS behind a smooth transparent foil tensioned withina frame and arranged at an angle of approximately 45 degrees to thefloor. The lens would be positioned in the central point of the screen,corresponding approximately to the central point of the audience. Areflective projection screen (RP or FP) may be arranged on the floor orthe ceiling of the filming studio. The projection screen emits the ASFimage/s in the same way as the conventional RP screen. However thepositioning of the filming camera lens central rather than peripheral tothe audience field area significantly improves referencing for betterpositional reflexes and eye level contact between audience participantand SS.

This final arrangement is the preferred set up to be used for a TPmeeting room experience. In this particular embodiment, a foil tensionedwithin a frame is arranged at 45 degrees to the floor, approximately inthe center of the room, almost cutting the room in half. A projector isarranged as shown in FIG. 28 to project a virtual image upon a stagefrom a remote SF. The stage is framed on 3 sides by dark covered wallsand ceiling. A suitable TP camera is arranged at one end of the room,upstage of the foil, in the same field area as the virtual images, toface the live talent/audience participants. The far wall facing thecamera can be covered either with black material drape or, in shortthrow distances (where lighting required to illuminate live stage talentwould otherwise spill onto the wall causing the black material to becomegrey), a blue-screen/green-screen back drop and floor arrangement ispreferred. This is because if the black curtain is over lit such that itturns grey, the clarity of the virtual image is compromised,particularly around the image outline—a fuzziness which renders thevirtual image less realistic.

The preparation of a blue screen or green screen room does presentchallenges in time preparation and practical considerations during use.Firstly the time and expense to cover the relevant wall, ceiling andfloor surfaces could in some cases be burdensome. Moreover the intensecoloured surroundings can be distracting to the live talent Performersor Audience Participants. A further practical consideration is that if ascreen is painted blue then any subject matter in whole or part thatincludes the colour blue when filmed will appear invisible (ortransparent if projected using foil). Thus an item of blue clothing orblue colouring on an item being filmed for presentation will lose formand realism. The same issues apply to a green screen room.

Where this solution is found to be expensive or impractical anothersolution would be to create a virtual blue screen environment byarranging along the facing wall a grey silvered curtain which whenfilmed with a camera lens ringed by a circle of tightly formed blue LEDlights is designed to automatically key out the background and isolatethe subject matter in the foreground. The curtain is unobtrusive andworks as a means of keying out foreground subjects (ideal in peppersghost projection) even when subjected to a fair amount of light spillfrom high powered filming lights. Blue or green items of clothing filmperfectly satisfactorily.

This solution is particularly preferred when the filming room haslimited space, say less than 12 m depth.

A modest lighting arrangement lights the room upstage of the foil toprovide the illusion of depth for the virtual image. A more substantiallighting rig is arranged downstage of the foil to correctly illuminatethe live talent being filmed. This lighting rig may be free standing.Preferably the lights will be retained by a truss frame, possibly anextension of the foil truss. Two rooms arranged in this way with camerasand video sources networked in a TP configuration would provide anexperience akin to a giant telephone box featuring life size, life likereal time interactive communication between at least two remote parties.

The final reference camera positions are those to provide necessaryreference for interaction between live and virtual stage Performers andCompares. At least one camera and display screen is required for eachstage. The object of these cameras is to provide accurate positionalreference of the on stage talent movement. Their deployment has beendetailed earlier. The frame rate/data rate/encode of these signals maybe subject to the greatest compression if limited Internet bandwidthrequires them to be so, since these images are not viewable by theaudience. The cameras maybe positioned anywhere convenient, butdesirably they will be positioned at eye level. The screen display maycomprise a conventional monitor panel, rear projection screen, anoptically clear reflective foil or a foil parabolic mirror. Blackcurtains would be the most preferable backdrop to filming each livestage talent and in certain circumstances the silvered grey screenarrangement could be used.

The immersive experience for TP is completed by selecting, arranging andprogramming the correct lighting configuration. The key areas to beilluminated are the stage and backdrop on the BS (to provide depth andcontrast for live and virtual talent), the live talent upon the stage(for live audience view and TP camera for SF to SS), the live audienceat BS and the live presenter at SS.

A range of lighting is available for each application. Overallobjectives are to present an environment of immersive ambience to thevenue overall, a compelling mixture of colour and contrast on stage andcorrectly illuminated live talent on stage and audiences for sharp,realistic SFs and ASFs.

Conventional TP lighting is able to satisfactorily illuminate seatedlive participants for the camera lenses to relay sharp HD images upon HDmonitor screens. To light the live Performer in the Source Studio a moreconsidered approach must be taken. The human figure for the purpose oflighting is essentially divided into two main parts (head to waist,waist to feet) but adds left and right control for the back of the head,face (shadow fill) and hair fill as separate elements. Lighting a humanfigure for a ‘holographic’ effect needs to fulfill the followingcriteria:

Be bright enough to capture subject detail in a uniform manner withoutdark spots (otherwise image becomes invisible or disappears) or overlybright spots (image bleaching). The lighting should pick out differingtextures as well as cast shadow across the subject accentuating form andthe passage of light movement across the subject. Back light should forma rim around the subject outline for maximum image sharpness.

The colour temperature of the lighting upon performer should whenappearing as the virtual Performer on the BS yield a skin tone that isnatural and matches as close as possible the hue and colour temperatureof the skin tones of similar skin types performing as live talent uponthe BS.

Lighting for 4-5 m Film Studio would Desirably Include the FollowingComponents:

5×ETC Source4 (50° or 25-50 zoom) ellipsoidal spots* on high (14 ft)stands

4×ETC Source4 (50° or 25-50 zoom) ellipsoidal spots* on lowboys/turtles(lens height to match height of studio staging)

1×ETC Source4 (36° or 25-50 zoom) ellipsoidal spot* flown/hung as centrebacklight * Alternatives accepted: any 750 w HPL or 1 kW/1.2 kW tungstenprofile/ellipsoidal spot with beam angles as specified

2×4 Bank 4 ft KinoFlos (tungsten tubes) on goalpost over studio staging

2×2 kW Fresnels on regular stands each through 4′×4′ diffusion frame(F2/1/2 Diff)

2×650 w Fresnels on lowboys/turtles

A selection of large and small flags and Charlie bars.

Stands, knuckles, etc for above

Various black/white poly/foam core sheets and support

ND0.3 and ND0.6 filter

Hampshire Frost, F1/1/4 Diff and F2/1/2 Diff filter

4-9 ways of 2 kW dimming if available

The key to lighting the live talent on-stage is to have the ability tomatch the colour temperature, intensity and angles of the lighting forthe person that is being transmitted to the live stage. There are anumber of ways which this can be achieved. One option is to use a numberof static lights (generics) to firstly be rigged at the correct anglesto light the live talent. These lights would then need to be colourcorrected with gel to match the colour temperature of the holographicimage. Another method would be to use moving lights to light the livetalent.

The use of moving wash lights would make adjustments easier to light thelive talent as one of the major problems with lighting using genericlanterns is that as you bring the intensity of light on the live talentdown the colour temperature it emits will change and there will be agreater mismatch in colour temperatures. If moving lights are used theymaintain a constant colour temperature as their intensities are reducedthus making the match a lot easier. Also the moving wash lights have anintegrated colour mixing system using cyan, magenta, yellow andoccasionally cto (colour temperature orange). These effects make itparticularly suitable to balance the colour temperature between the liveand the holographic talents.

Another element of the lighting for the live stage element of the TP isthe importance of creating the illusion of depth on the stage so thatthe holographic talent appears to stand out from the back drop andtherefore becoming more lifelike. Again it is possible to use genericlighting to perform this function. I.e. up-lighting the backdrop of thestage with floor mounted par cans, making sure that none of these lightsilluminate the area behind the holographic talent as this lighting willoverpower the holographic projection and take away from the overalleffect. Care needs to be taken to also ensure that the lighting level isconsistent throughout the viewing angle of the system. To make this taskeasier again the use of moving head wash and spot lights can be usedwith the addition of LED battons and par type fixtures. The advantage tousing moving lights and LED technology is that you can alter theintensity, position, colour and texture on the backdrop to avoid theposition of the holographic talent in the live environment. The LEDlighting can provide a static colour changing facility with the abilityto alter the intensity; this again performs the same function of themoving lights.

The fighting of the live audience element of the TP is another importantelement to the TP experience. The most practical method is to suspend anumber of trusses above the audience in such a position that you canadequately front light and back light the audience at a reasonable levelso that the camera relay can pick out people both for a close up camerashot and a wide camera shot. You could also use follow spots toindividually light a member of the audience that the holographic talenthas a specific interest in (i.e. questions and answers). An even morepractical method would be to use moving spotlights suspended on trussesin a plurality of points to provide the same style of lighting as thefollow spots.

Stage lighting preferably embodies the following basic equipment:

Lighting Desk

2×Mac 500's

2×Mac 600's

2×Mac 300's

1×Pixelline

1×32 amp 3 Phase power distribution box with 6×16 amp 1 phase outputs

8×16 amp-16 amp Cables

4×16 amp Splitters

8×10 m 3 pin XLR Cables

2×10 m 5 pin DMX Cables

4×5 m 5 pin DMX Cables

Basic Spec/Black Figure Version

Camera: Sony HDW750 or HDW790 or F900/3 or R shooting 1080/50i or1080/60i/59.94i as required

Standard lens

Matte box

Polarising filter

Tripod (tall and short legs)

Battery kit/mains kit

20 inch (or larger) HD CRT monitor

Chroma background rig as suggested by studio

Notes for Black-Clothed Subject:

Suggest blue/green screen as figure has to appear as solid (unlit)silhouette at start

Add 2×8×4 white poly for modelling

Add 2×2 kW Fresnels for above

Add memory lighting.desk and dimming for all ‘subject’ lights (notchrome screen rig)

Peppers Ghost—Filming Guidelines

General

The aim of a Musion live action shoot is to produce a life-like, highdefinition video image that best fulfills the technical and aestheticrequirements of the Musion presentation system. It should also capturethe most appropriate content for the project. Relevant elements include:a true ‘black’ background; effective lighting to enhance the projectedimage; correct colour balance; minimum motion blur without astrobing/shuttered look; correct camera height to represent the audienceeyeline; effective ‘costume’ control to suit the talent, content andknown benefits to the projected image; realistic interaction withgraphics or other elements; make full use of the elements known to‘work’ in a Musion presentation.

The methods of producing a suitable image to pre-record or for live TPtransmission are fundamentally similar.

Studio

Booking a professional studio with its ancillary facilities and anexperienced crew will allow everyone to concentrate on the project andultimately produce a more satisfactory result.

Certain subjects will have specific requirements—for instance, a carshoot—but most are a variation on a number people standing on a stage.The following assumes a basic scenario of a 4-5 m wide stage.

To create a good black background it is essential to have the maximumdistance between the ‘stage’ and the background black drapes. Around 10m works well without having to introduce too much negative Master Blackcontrol (black crush) via the camera menu settings.

The barrel distortion (exaggerated perspective) of wide angle lensesshould be avoided, thus a camera/stage distance of around 10 m needs tobe achieved. This allows a 22-25 mm lens (⅔″ chip) to cover a 4 m wideaction area and the distortion is reduced.

To replicate an average audience eyeline, the camera generally has to beset very low. The result is that, at the above distances, the backgroundneeds to be 8.5 m high.

To set the rim lights at the ideal angle, at least 2 m space either sideof the stage is needed.

Ideally, therefore, the studio dimensions would be around 20 m (l)×10 m(w)×9 m (h). Black drapes to back wall. Black drapes to floor whereseen. Semi-matt (e.g. ‘Harlequin’ dance floor) or high gloss surface tostage (a stylistic decision).

Sufficient Power Supply for Lighting.

A Steeldeck stage or similar gives the subject a spacial boundary towork within and should match the dimensions of the show stage or theprojected area whichever is smaller. The projection limits should beexplained to the subject and markers set for him to see, but are notvisible to the camera. Although the height of the studio stage need notbe the same as the show stage, the difference is an essential figure incalculating the height of the camera. The stage also avoids having toset the camera on the studio floor to achieve the correct height.

As most shoots will involve the recording of sound, a properly soundinsulated and acoustically treated studio should be used. Bear in mindthat the sound will be reproduced at a high level in the presentationand every extraneous sound will be heard. A professional sound recordistshould use high quality microphones to record via a boom orpersonal/radio mic as appropriate.

Camera Equipment/Settings

Musion live action is currently shot in full High Definition(1920.times.1080). For recorded sequences, HDCam or HDCamSR is used andan HD-SDI signal provided for live transmissions (this is subject tocontinuing development and updating).

Cameras Commonly Used are:

Sony HDW-750P (HDW750 for 59.95i)/HDW790P (HDW790 for 59.94i)

Sony HDW-F900/R (all frame rates)

HDC-F950 (all frame rates to HDCamSR)

Interlaced frame rates (50i/59.94i) are used as this produces the mostlife-like motion. Progressive scanned images look too ‘shuttered’ andfilm-like.

A 1/60th second shutter is used to reduce motion blur without shutterartifacts becoming too noticeable. Higher shutter speeds can be testedwith slow moving subjects.

Use a small amount of black crush (Master Black to −1 or −2) to bringthe black threshold above the video noise level. Turn the monitorbrightness up to reveal the noise and the effect of the adjustment.Avoid crushing shadow areas of the subject.

Standard lenses (e.g. Canon HJ21 or 22) are favoured over wide anglezooms. Prime lenses can also be used as long as the camera/stagedistance is not compromised by their fixed focal length. The effects oftoo wide a lens angle include: enlarged hands when gesturing; enlargedhead or legs, depending on camera height; the appearance of growing orshrinking with movements towards or away from camera; bowed floor line.At 10 m, a lens of 22-25 mm over a 4 m width substantially reducesdistortion.

A high quality Polarising camera filter is used to control speculareflections from either the floor or subject (orientation byexperiment).

A full set of tripod legs (tall, baby and HiHat) should be available asthe camera height may be very low depending on studio stage height andaudience eyeline. As the shot is a lock-off, the head needs to haveeffective pan and tilt locks.

Monitors

The quality of HD photography, especially for large screen projection,needs to be assessed with a very high degree of accuracy. A hair out ofplace or a marginal error in lens focus will not be apparent on a 9 inchLCD monitor. The essential parameters of lighting level and balance,focus and fine detail need to observed on at least a 20-inch HD-SDI CRTmonitor (ideally Grade 1 or 2). Current LCDs do not have the resolutionor contrast range to be a reliable guide to the projected image. Thesmallest details must be addressed: hair, makeup, clothes, shoes, propsas nothing is hidden when projected life-size with a black background.

Lighting

Although it is recommended that the lighting follows the basicproscribed plan, the resulting ‘look’ should be created by the DoP toreflect the director's ideas and the concept of the project as discussedat the pre-production stage. Certain guidelines should, however, betaken into account.

Slightly exaggerated back/rim light gives the projected image enhancedbrightness and sharpness. It also encourages the perception of a 3Dimage.

A ‘rounded’ light technique works well. Reducing the front/fill levelcompared to the side and rim light emphasises the third dimension.

A tightly slotted ‘eye light’ near to the camera line will lift deep-seteyes without over-filling the body.

Dark and glossy hair can be lifted by adjusting the height and positionof the overhead KinoFlo fixtures.

The floor level Fresnel spots can help to fill deep shadows caused byloose fitting clothes or unbuttoned jackets.

If the lighting is too flat, it gives the impression of cardboard cutout figures (distinctly 2D).

Video acquisition does not like highlights—ensure that bright areasmaintain detail and do not burn out. Avoid using DCC unless verycarefully controlled.

Pay particular attention to the legs and feet of the subject. Make sureboth are clearly defined—even to the extent of insisting that the shoesare changed to make them visible.

In certain—mainly music—situations, the design may require additionalcolour washes. These are most effective as rim and side light using alimited range of distinctive colours. To make a substantial impact, theintensity of the heavily coloured sources must be sufficient to showabove the existing rim light. PAR64 battens are an effective, ifunsubtle, supplement to the lighting rig.

Eyeline

It is important to understand that the subject is photographed in 2D. Asthe eyes of a painted subject ‘follow you around the room’, so thefilmed subject looking straight into the camera lens makes eye contactwith everyone in the audience. Contributors often have to be coachedinto not scanning the room as they would in a live situation.

It is essential to get the relative eyeline height correct. If it iswrong, the subject will appear to the key audience members to be leaningbackwards or forwards.

A formula allows the camera shooting height to be determined. This takesinto account the show stage height, the selected audience eyeline, thestudio stage height and the relative distances of the camera andaudience.

REFERENCE NUMERAL KEY

FIG. 1

-   10 Microphone-   12 Camera-   14 Monitor/Display-   16 Speaker-   18 Codec Box with Audio and Video Inputs and Audio and Video Outputs-   20 Transmission over Internet, Satellite, or Radio Waves-   22 Codec Box with Audio and Video Inputs and Audio and Video Outputs-   24 Microphone-   26 Camera-   28 Monitor/Display-   30 Speaker    FIG. 2-   32 Codec Box-   34 Filters, Limiters, Gates, Compression-   36 Filters, Time-Based Corrector-   38 EQ Delay-   40 Time-Based Corrector-   42 Codec Encoder-   44 Codec Decoder-   46 Multiplexer for Multistrand Transmission-   48 Signal Feed Connection-   50 Demultiplexer for Multistrand Transmission-   52 Bidirectional Communication Connection to other Codec Boxes    (Example Connector: TCPIP)    FIG. 4-   54 Silver Drape Line for Chroma Keying-   56 Reference Monitor for Displaying Graphic/Documents or using    Picture-in-Picture in Projector-   58 Optional Camera-   60 Person-   62 Stage-   64 Projector-   66 Projection Screen-   67 Light emitted by Projector-   68 Foil-   70 Hologram-   72 Black Draping (MATT) to stop flaring camera from foil-   74 Monitor (Small Monitor to add Picture-in-Picture next to Film    Talent)-   76 Stage-   78 LED Ring for Cinema Keying-   79 Black Drape-   80 Camera    FIG. 23-   82 Black Drapeline-   84 Perceived Hologram-   86 Stage-   88 Leg-   90 Video Wall on its back-   92 Musion EyeLiner Foil Tensioned in Frame-   94 Front Mask (Top)-   96 Rigging Point-   98 Front Mask (Bottom)-   99 Sightline-   100 Person    FIG. 24-   102 Stage-   104 Person-   106 EyeLine-   108 EyeLiner Foil-   110 Camera Shot Lines-   112 Black Material on floor to stop glare/flare of Foil and Camera    Lens/Image-   114 Heads Up Display-   116 Image from Heads Up Display-   118 Pepper's Ghost Image from Heads Up Display-   120 Camera    FIG. 25-   122 Leg-   123 Leg-   124 Parabolic Mirror-   126 Masking-   128 Mask-   130 Adjustable Arm for moving Display Device up/down, left/right,    in/out, to focus image in required position-   132 Display Device: Monitor (CRT/LCD/Plasma); Projection (Screen)-   134 Perceived Hologram-   136 Sightline-   138 Mask-   140 Person    FIG. 26-   142 Person-   144 EyeLine-   146 Camera Shot Lines-   148 Camera-   150 Perceived Hologram-   152 Projection Screen-   154 Mask-   156 Foil-   158 EyeLine-   160 Mask-   162 Mask-   164 Mirror-   166 Projector-   168 Person-   170 Light from Projector reflected by Mirror    FIG. 27-   172 Camera-   174 Black Drapes-   176 Perceived Hologram-   178 Side Monitor-   180 Camera-   182 Heads Up Display-   184 Foil-   186 Light from Projector-   188 Lights-   190 Light from Lights-   192 Light from Lights-   194 Optional Ambiance Boom Mics-   196 Optional Ambiance Boom Mics-   198 Lights-   200 Light from Lights-   202 Side Speakers-   204 Mask-   205 Optional Mics on Front Mask-   206 Mask-   208 Seat-   210 Optional Mics on Back of Seat-   212 Optional Mic on Back of Seat-   214 Seat-   216 Seat-   218 Boom Mic Operator-   220 Boom Mic-   222 Camera-   224 Lights    FIG. 28-   226 Black Silver Chroma Keying Drape-   228 Light-   230 Light from Light-   232 Light from Light-   234 Light-   236 Optional Side Reference Monitor-   238 Optional Side Reference Camera-   240 Projection Screen-   242 Seat-   246 Backlight for Talent-   248 Stage-   250 Person-   252 Light from Projector-   254 Lights mounted to Frame-   256 Lights mounted to Frame-   258 Frame with Foil tensioned in it-   260 Projector-   262 Black Mat Floor Covering-   264 Stage-   266 Light-   268 Optional Extra Keying Light-   270 Light from Light-   272 Black Drapes-   274 Seat-   276 Camera-   278 Light-   280 Blue/Green Lens for Chroma Keying

GENERAL

The aim of a Musion live action shoot is to produce a life-like, highdefinition video image that best fulfils the technical and aestheticrequirements of the Musion presentation system. It should also capturethe most appropriate content for the project. Relevant elements include:a true ‘black’ background; effective lighting to enhance the projectedimage; correct colour balance; minimum motion blur without astrobing/shuttered look; correct camera height to represent the audienceeyeline; effective ‘costume’ control to suit the talent, content andknown benefits to the projected image; realistic interaction withgraphics or other elements; make full use of the elements known to‘work’ in a Musion presentation.

The methods of producing a suitable image to pre-record or for livetransmission are fundamentally similar.

Studio

Booking a professional studio with its ancillary facilities and anexperienced crew will allow everyone to concentrate on the project andultimately produce a more satisfactory result.

Certain subjects will have specific requirements—for instance, a carshoot—but most are a variation on a number people standing on a stage.The following assumes a basic scenario of a 4-5 m wide stage.

To create a good black background it is essential to have the maximumdistance between the ‘stage’ and the background black drapes. Around 10m works well without having to introduce too much negative Master Blackcontrol (black crush) via the camera menu settings.

The barrel distortion (exaggerated perspective) of wide angle lensesshould be avoided, thus a camera/stage distance of around 10 m needs tobe achieved. This allows a 22-25 mm lens (⅔″ chip) to cover a 4 m wideaction area and the distortion is reduced.

To replicate an average audience eyeline, the camera generally has to beset very low. The result is that, at the above distances, the backgroundneeds to be 8.5 m high.

To set the rim lights at the ideal angle, at least 2 m space either sideof the stage is needed.

Ideally, therefore, the studio dimensions would be around 20 m (l)×10 m(w)×9 m (h). Black drapes to back wall. Black drapes to floor whereseen. Semi-matt (e.g. ‘Harlequin’ dance floor) or high gloss surface tostage (a stylistic decision).

Sufficient Power Supply for Lighting.

A Steeldeck stage or similar gives the subject a spacial boundary towork within and should match the dimensions of the show stage or theprojected area whichever is smaller. The projection limits should beexplained to the subject and markers set for him to see, but are notvisible to the camera. Although the height of the studio stage need notbe the same as the show stage, the difference is an essential figure incalculating the height of the camera. The stage also avoids having toset the camera on the studio floor to achieve the correct height.

As most shoots will involve the recording of sound, a properly soundinsulated and acoustically treated studio should be used. Bear in mindthat the sound will be reproduced at a high level in the presentationand every extraneous sound will be heard. A professional sound recordistshould use high quality microphone to record via a boom orpersonal/radio mic as appropriate.

Camera Equipment/Settings

Musion live action is currently shot in full High Definition(1920×1080). For recorded sequences, HDCam or HDCamSR is used and anHD-SDI signal provided for live transmissions (this is subject tocontinuing development and updating).

Cameras Commonly Used are:

Sony HDW-750P (HDW750 for 59.95i)/HDW790P (HDW790 for 59.94i)

Sony HDW-F900/R (all frame rates)

HDC-F950 (all frame rates to HDCamSR)

Interlaced frame rates (50i/59.94i) are used as this produces the mostlife-like motion. Progressive scanned images look too ‘shuttered’ andfilm-like.

A 1/60th second shutter is used to reduce motion blur without shutterartifacts becoming too noticeable. Higher shutter speeds can be testedwith slow moving subjects.

Use a small amount of black crush (Master Black to −1 or −2) to bringthe black threshold above the video noise level (turn the monitorbrightness up to reveal the noise and the effect of the adjustment).Avoid crushing shadow areas of the subject.

Standard lenses (e.g. Canon HJ21 or 22) are favoured over wide anglezooms. Prime lenses can also be used as long as the camera/stagedistance is not compromised by their fixed focal length. The effects oftoo wide a lens angle include: enlarged hands when gesturing; enlargedhead or legs depending on camera height; the appearance of growing orshrinking with movements towards or away from camera; bowed floor line.At 10 m, a lens of 22-25 mm over a 4 m width substantially reducesdistortion.

A high quality Polarising camera filter is used to control speculareflections from either the floor or subject (orientation byexperiment).

A full set of tripod legs (tall, baby and HiHat) should be available asthe camera height may be very low depending on studio stage height andaudience eyeline. As the shot is a lock-off, the head needs to haveeffective pan and tilt locks.

Monitors

The quality of HD photography, especially for large screen projection,needs to be assessed with a very high degree of accuracy. A hair out ofplace or a marginal error in lens focus will not be apparent on a 9 inchLCD monitor. The essential parameters of lighting level and balance,focus and fine detail need to observed on at least a 20-inch HD-SDI CRTmonitor (ideally Grade 1 or 2). Current LCDs do not have the resolutionor contrast range to be a reliable guide to the projected image. Thesmallest details must be addressed: hair, makeup, clothes, shoes, propsas nothing is hidden when projected life-size with a black background.

Lighting

Although it is recommended that the lighting follows the basicproscribed plan, the resulting ‘look’ should be created by the DoP toreflect the director's ideas and the concept of the project as discussedat the pre-production stage. Certain guidelines should, however, betaken into account.

Slightly exaggerated back/rim light gives the projected image enhancedbrightness and sharpness. It also encourages the perception of a 3Dimage.

A ‘rounded’ light technique works well. Reducing the front/fill levelcompared to the side and rim light emphasises the third dimension.

A tightly slotted ‘eye light’ near to the camera line will lift deep-seteyes without over-filling the body.

Dark and glossy hair can be lifted by adjusting the height and positionof the overhead KinoFlo fixtures.

The floor level Fresnel spots can help to fill deep shadows caused byloose fitting clothes or unbuttoned jackets.

If the lighting is too flat, it gives the impression of cardboard cutout figures (distinctly 2D).

Video acquisition does not like highlights—ensure that bright areasmaintain detail and do not burn out. Avoid using DCC unless verycarefully controlled.

Pay particular attention to the legs and feet of the subject. Make sureboth are clearly defined—even to the extent of insisting that the shoesare changed to make them visible.

In certain—mainly music—situations, the design may require additionalcolour washes. These are most effective as rim and side light using alimited range of distinctive colours. To make a substantial impact, theintensity of the heavily coloured sources must be sufficient to showabove the existing rim light. PAR64 battens are an effective, ifunsubtle, supplement to the lighting rig.

Eyeline

It is important to understand that the subject is photographed in 2D. Asthe eyes of a painted subject ‘follow you around the room’, so thefilmed subject looking straight into the camera lens makes eye contactwith everyone in the audience. Contributors often have to be coachedinto not scanning the room as they would in a live situation.

It is essential to get the relative eyeline height correct. If it iswrong, the subject will appear to the key audience members to be leaningbackwards or forwards.

A formula allows the camera shooting height to be determined. This takesinto account the show stage height, the selected audience eyeline, thestudio stage height and the relative distances of the camera andaudience.

The invention claimed is:
 1. A telepresence system, comprising: aprojector for generating an image, a projection screen for receiving theimage generated by the projector and generating a reflected image; afoil for receiving the reflected image generated by the projectionscreen and generating and directing a partially reflected image towardan audience, the partially reflected image being perceived by theaudience as a virtual image or hologram located on a viewing stage; anda camera for filming an individual through the foil, the camera beinglocated on a camera side of the foil and positioned adjacent to theviewing stage, the individual being located on an individual side of thefoil and positioned on a filming stage, and using images captured by thecamera to provide a heads up display, the heads up display adapted togenerate an image to be received by the foil for generating a pepper'sghost virtual image invisible to the audience.
 2. A telepresence systemaccording to claim 1 wherein the camera arranged for filming anindividual through the foil is arranged to film the audience and thevirtual image is arranged to provide audience feed back to a performer.3. A telepresence system according to claim 2 wherein the virtual imageis generated such that it provides line of sight guidance for theperformer that is referentially accurate in terms of eyeline.
 4. Atelepresence system according to claim 3 wherein the heads-up displaycomprises a mechanically moveable video screen/s upstage to the foil,the foil arranged such that light emitted from the video screen isreflected therefrom; and the video screen being arranged to project animage such that light forming the image impinges upon the foil to createthe virtual image, the system adapted such that the video screen/s aremechanically moveable stage left or right along a truss using rollersand the angle of the screen is adjustable to position the virtual imagehigher or lower around the audience.
 5. A telepresence system accordingto claim 2 in which the system includes an audience management systemadapted to highlight in a way recognisable to the camera the position ofan audience member and a program control of the camera is adapted todirect and focus it predominantly on the audience member.
 6. Atelepresence system according to claim 5, in which the highlight used bythe audience management system comprises any of an infrared signal, anultraviolet signal or a black and white patterned bar code.
 7. Thesystem of claim 1, wherein: the projector is located on a floor andoriented so that the image is directed upward; and the projection screenis located on a ceiling above the projector.
 8. The system of claim 1,wherein the foil is positioned adjacent to the projector and theprojection screen and oriented at an angle of 45 degrees with respect tothe projection screen.
 9. The system of claim 1, wherein the cameraincludes an LED ring for chroma keying.
 10. The system of claim 1,further comprising black draping adjacent to the projector and below thefoil for reducing flaring in the camera caused by the foil.
 11. Thesystem of claim 1, further comprising a monitor for generating a picturein picture image, the monitor being positioned below the foil andoriented so that the picture in picture image is directed onto the foiland reflected toward the camera.
 12. The system of claim 1, furthercomprising a reference monitor adjacent to the individual for displayinggraphics and documents to the individual during filming.
 13. The systemof claim 1, further comprising a reference camera adjacent to theindividual for generating a reference image of the individual.
 14. Thesystem of claim 1, further comprising silver or black draping adjacentto the filming stage for chroma keying.
 15. The system, of claim 1further comprising: a frame for providing support to the projector andfoil, the projector being connected to the frame and the foil beingconnected to the frame using a tensioning device, the tensioning devicebeing capable of variably adjusting tension along at least one edge ofthe foil so that the foil remains substantially flat and smooth.
 16. Thesystem of claim 15, wherein: the projector is connected to an upperportion of the frame and oriented so that the image is directed downwardtoward the projection screen; and the projection screen is located belowthe projector.
 17. The system of claim 15, further comprising drapespositioned so that the projection screen cannot be seen by the audience.18. The system of claim 15, further comprising a mirror connected to theupper portion of the frame adjacent to the projector for reflecting theimage generated by the projector onto the projection screen and whereinthe projector is oriented to direct the image toward the mirror.
 19. Thesystem of claim 15, wherein: the projector is connected to a lowerportion of the frame and oriented so that the image is directed upwardtoward the projection screen; and the projection screen is positionedabove the projector.
 20. The system of claim 19, further comprising amirror connected to the frame adjacent to the projector for reflectingthe image generated by the projector onto the projection screen andwherein the projector is oriented to direct the image toward the mirror.21. The system of claim 15, wherein: the projection screen is locatedadjacent to a lower portion of the frame; and the projector ispositioned below the projection screen and oriented so that the image isdirected upward toward the projection screen.
 22. The system of claim21, further comprising a mirror positioned adjacent to the projector forreflecting the image generated by the projector onto the projectionscreen and wherein the projector is oriented to direct the image towardthe mirror.
 23. The system of claim 15, wherein: the projection screenis located adjacent to an upper portion of the frame; and the projectoris positioned above the projection screen and oriented so that the imageis directed downward toward the projection screen.
 24. The system ofclaim 23, further comprising a mirror positioned adjacent to theprojector for reflecting the image generated by the projector onto theprojection screen and wherein the projector is oriented to direct theimage toward the mirror.
 25. The system of claim 1 wherein the camera isarranged for recording the reflected image and a person located on thefilming stage.
 26. The system of claim 25, wherein the Pepper's Ghostimage is located between the foil and the camera.
 27. The system ofclaim 25, further comprising black material positioned so that the blackmaterial reduces or eliminates glare caused by the foil in a camera lensincluded with the camera.
 28. The system of claim 25, wherein thedisplay is located above the foil and the black material is positionedbelow the foil.